MPR Reference No.: 6215-400 



MATHEMATICA 

Policy Research, Inc. 



School Principals’ 
Perspectives on the 
Passport to Teaching 



June 28, 2006 



Steven Glazerman 
Christina Tuttle 
Gail Baxter 



Submitted to: 



Submitted by: 



American Board for Certification of 
Teacher Excellence 



Mathematica Policy Research, Inc. 
600 Maryland Ave. S.W., Suite 550 



1225 19“^ Street, N.W., Suite 400 
Washington, DC 20036 
Telephone: (202) 261-2620 



Washington, DC 20024-2512 
Telephone: (202) 484-9220 
Facsimile: (202) 863-1763 



Project Officer: 
Joshua Boots 



Project Director: 

Steven Glazerman 



ACKNOWLEDGMENTS 



Several people contributed to the research documented in this report. We could not have 
done this work without the gracious cooperation of the principals in our sample. At the 
American Board for Certification of Teacher Excellence (ABCTE), Joshua Boots, David Saba, 
and William Schimmel provided comments, data, and advice. 

The data collection team at Mathematica Policy Research included Brian Roff, Milena 
Rosenblum, Eaurie Bach, and Jessy Nazario, who made the survey run smoothly. Duncan 
Chaplin provided useful guidance and input at every stage. John Deke carefully read the draft 
report and provided helpful comments. Elizabeth Seif provided able research assistance. Carol 
Soble edited the report, and Sharon Clark prepared it for publication with assistance from Donna 
Dorsey. 




CONTENTS 



Chapter Page 

I PASSPORT TO TEACHING 1 

A. THE PASSPORT PROGRAM 1 

B. THE EONGITUDINAE STUDY AND THIS REPORT 2 

II DATA AND METHODS 4 

A. SURVEY DESIGN 4 

B . VAEIDITY OE THE PRINCIPAE SURVEY 4 

C. SAMPEING ERAME, SAMPEE DESCRIPTION, 

AND NONRESPONSE 6 

III EINDINGS 10 

A. HIRING DECISIONS AND PERCEPTIONS OP ABCTE 10 

B . PERPORMANCE RATINGS 13 

C. UNDERSTANDING THE PERPORMANCE RATINGS 17 

IV CONCEUSIONS 21 

A. SUMMARY OP EINDINGS 21 

B . EIMITATIONS OP THE STUDY 22 

REPERENCES 23 



iii 




TABLES 



Table Page 

II. I RESPONSE RATES BY SUBGROUP 7 

11.2 CHARACTERISTICS OE RESPONDENTS AND 

NONRESPONDENTS (PERCENTAGES) 8 

11.3 EXPERIENCE EEVEE OE PASSPORT TEACHERS 9 

III. I EACTORS REEATED TO HIRING DECISIONS II 

111.2 EEEECTIVENESS OE PASSPORT HOEDERS AS RATED 

BY THEIR PRINCIPAES 14 

111.3 PRINCIPAE RATINGS OE OVERAEE PEREORMANCE 

AND CONEIDENCE IN TEACHERS AND ABCTE 16 

111.4 THE EEEECT OE PRINCIPAE AND TEACHER 

CHARACTERISTICS ON OVERAEE RATINGS 19 

111.5 THE EEEECT OE COMPONENTS ON OVERAEE 

EEEECTIVENESS RATING 20 



IV 




I. PASSPORT TO TEACHING 



A. THE PASSPORT PROGRAM 

State governments use a wide variety of requirements to regulate entry into the teaching 
profession. They vary in the degree to which they require prospective teachers to complete 
coursework, accumulate student teaching experience, and perform on written examinations 
before they are granted entry into the classroom. The goal of the licensure regulations should be 
to ensure minimum quality, but education researchers have not succeeded in isolating measures 
that cost-effectively and reliably predict teacher performance. 

In recent years, the American Board for Certification of Teacher Excellence (ABCTE), a 
new organization funded by the U.S. Department of Education, has devised a primarily 
examination-based approach called the Passport to Teaching to address this problem. With the 
Passport program, ABCTE set out to develop rigorous and comprehensive examinations in 
professional teacher knowledge and subject matter that would identify people competent to enter 
the teaching profession in any state regardless of the individual’s method of preparation. By 
encouraging states to adopt the Passport as an acceptable route into the classroom, ABCTE 
hopes to remove barriers to entry such as student teaching requirements or specific course 
requirements that might otherwise deter qualified candidates. Eor example, the Passport could 
create opportunities for career changers, for veteran teachers with no formal credential, and for 
teacher candidates with nontraditional preparation to enter or remain in teaching. 

To date, five states have recognized the Passport to Teaching. Those states are Idaho, 
Elorida, Pennsylvania, New Hampshire, and Utah. One challenge to the Passport program is the 
opposition by teacher-educators who see the program as a threat to traditional, degree-granting 
teacher preparation programs, which offer the courses and student-teaching opportunities that 
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would be eliminated as requirements if the Passport were accepted for certification. Another 
major impediment to more widespread adoption of the Passport is the need for more rigorous and 
objective evidence on the ability of the Passport program to identify high-quality or even 
minimal-quality teachers. 

B. THE LONGITUDINAL STUDY AND THIS REPORT 

To help address the need for rigorous research evidence, ABCTE obtained a grant from the 
U.S. Department of Education to contract with Mathematica Policy Research, Inc. (MPR) to 
conduct an independent, five-year longitudinal evaluation of American Board certification 
programs. In a recent report (Glazerman and Tuttle 2006), we described the early progress of the 
Passport program and the plans for evaluating both the Passport and a newer certification 
program for veteran teachers. 

This report is the first in a series of reports from that evaluation study. Here, we provide 
some early evidence on the performance of American Board-certified teachers based on our 
recent survey of their supervisors. The study pertains to the first cohort of teachers to be 
certified through the Passport to Teaching program and begin teaching in 2005-2006. Nearly all 
of the supervisors we interviewed for this report were school principals; therefore, we refer to the 
telephone survey as the principal survey. The main goal of the principal survey was to learn how 
the Passport teachers’ direct supervisors judge their performance on the job. The performance 
ratings are subjective, but supervisors, unlike parents, fellow teachers, and students, have the job 
of observing and evaluating all aspects of a teacher’s performance and thus have the best 
perspective available. We also asked questions about the role of ABCTE certification in hiring 
that teacher. In a few cases, we interviewed a different individual to address hiring questions. 
Section B provides a detailed description of the data collection methods. 
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We found that principals typically rated Passport holders to be “as effective” as or 
“somewhat more effective” than “all other teachers [they had] observed in their career” on every 
dimension of job performance. A small number of Passport holders received low ratings 
(“much” or “somewhat” less effective), and a slightly larger number received “much more” 
effective ratings. Based on the ABCTE teachers they had supervised, principals were mixed on 
their appraisal of American Board certification, with the most common response that of 
uncertainty because of the limited experience with the program (typically just one teacher). 

It is important to consider this report in its context as the first of several on the American 
Board certification programs. We report here on the first cohort of Passport holders and expect 
that cohort to be somewhat idiosyncratic and smaller than future cohorts. Even for the current 
cohort, we are continuing to collect data on student achievement gains that will allow us to form 
more objective measures of performance to consider alongside the present findings. 

The next section discusses the data and methods used in this report. Section C presents the 
findings, and Section D concludes with a summary of the main findings and their implications 
for both policy and the ongoing research effort to understand ABCTE programs. 
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II. DATA AND METHODS 



A. SURVEY DESIGN 

We collected the data for the study by conducting a telephone survey of supervisors 
(principals) in spring 2006. This period corresponds to the end of the 2005-2006 school year, 
the first year in which it was possible to observe Passport-certified teachers in the classroom. 
The interview included a series of questions about the circumstances under which the Passport 
holder was hired, ^ the amount of time the respondent observed the teacher in the classroom, the 
respondent’s professional background, and the respondent’s ratings of the ABCTE teacher’s 
effectiveness. 

The effectiveness ratings were five -point scales that covered 11 dimensions ranging from 
instruction to collaboration and collegiality. We also asked respondents to rate the teacher’s 
performance overall and their confidence in the Passport to Teaching as a credential. To anchor 
the ratings, we asked principals to compare Passport teachers to all other teachers they have 
observed in their career. For novice Passport holders, we also asked principals to rate the 
Passport holders relative to all other novice teachers they have observed in their career. 

B. VALIDITY OF THE PRINCIPAL SURVEY 

A principal survey such as the one described here has advantages and disadvantages but 
overall was the most useful approach we considered for providing early evidence on teachers 
certified through the Passport program. The main disadvantage is that principal ratings are 
subjective and can say more about the person performing the rating than the teacher whose 

' We asked if the respondent was responsible for hiring the American Board teacher in question. In cases 
when the respondent was not the hirer, we identified the hirer and interviewed that person separately. The “hirer” 
and “observer” were usually the same person — the principal — except in five cases. 
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performance is rated. For example, when we ask principals to rate ABCTE teachers against 
others they have observed, each respondent has a different reference group. Some principals 
may have hired the Passport holder out of desperation after observing a string of poor teachers. 
Others may have a very high standard. This was a special concern for us because each 
respondent reported on just one teacher. Therefore, we had no way to calibrate scores 
statistically. 

We determined that subjective ratings are still useful because the principal, whose role is to 
observe and supervise the teacher in all aspects of the job, offers the best perspective available. 
And the reference group of the particular principals in our sample is relevant because we wish to 
know how ABCTE affects that status quo. Eurthermore, by asking about the principals’ 
background and the frequency and duration of their observations of the teacher, we can control 
some of the variation in factors that might affect any one principal’s perspective. Another 
advantage of principal ratings relative to measures that use standardized student test scores is that 
the ratings can be applied to all classroom teachers, including those in nontested subjects and 
grades. 

The research literature on the validity of principal ratings as a tool to measure teacher 
performance is not conclusive but does suggest that principal ratings predict achievement. A 
recent working paper by Jacob and Eefgren (2005) found that principals were able to predict 
which of their teachers produced the largest and smallest student achievement gains, although the 
principals could not distinguish among teachers in the middle of the performance distribution. 
Jacob and Eefgren cited earlier studies (Mumane 1975; Armor et al. 1976; Medley and Coker 
1987; Peterson 1987, 2000) that show correlations between principal ratings and other outcomes, 
albeit small ones. 
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Our principal survey findings are not intended to stand alone. Several measures of teacher 
effectiveness are needed to draw conclusions about the performance of Passport holders in the 
classroom. This report on principal ratings is intended to be a companion to future reports that 
incorporate analysis of growth in test scores in students of Passport and comparable non- 
Passport holders. 

C. SAMPLING FRAME, SAMPLE DESCRIPTION, AND NONRESPONSE 

We targeted the universe of teachers eligible to participate in the study. That is, we 
identified all Passport holders who were known to hold a K-12 classroom teaching position (52 
individuals) and contacted their schools or districts to identify appropriate respondents to the 
survey. The initial sample comprised 56 individuals, but 4 were deemed ineligible because their 
primary responsibility was not classroom teaching (2 were principals) or their status as 
classroom teachers could not be verified. 

We completed interviews with supervisors of 39 of the 52 eligible teachers, for a response 
rate of 75 percent. Typically, there was one Passport holder per school and hence one 
respondent per teacher. For one teacher, we have two sets of ratings because she split her time 
between two schools. For the analysis, we averaged her two sets of ratings. When we could not 
report fractional ratings — for example, in reporting the full distribution of scores — we included 
both principals’ ratings in the frequency count but adjusted the percentages by using weights so 
that each teacher was counted once. The weighting made a trivial difference in the percentages 
relative to the unweighted counts. 

The response rates were high across many subgroups, although some were higher than 
others. Table II. 1 shows the response rates by type of school, grade configuration, teaching 
assignment of the ABCTE teacher, and school location. The response rates were at least 70 
percent in traditional public, charter, and private schools. The response rate was higher for 
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elementary schools than for middle or high schools, as also reflected in the response rates by 
subject area, with principals of teachers assigned to elementary subjects responding at the highest 
rates. Sample members in Idaho, where most of the Passport holders were teaching, had the 
highest response rate, with response rates for sample members in other states lagging. 



TABLE II. 1 

RESPONSE RATES BY SUBGROUP 



Subgroup 


Eligible Cases 


Response Rate (percentage) 


Overall 


52 


75.0 


School Type 


Traditional public 


38 


73.7 


Charter 


4 


100.0 


Private 


10 


70.0 


Grade Eevel 


Elementary 


17 


88.2 


Middle 


5 


80.0 


High 


30 


66.7 


Teaching Assignment 


Elementary (all subjects) 


10 


90.0 


English/language arts (secondary) 


10 


60.0 


Mathematics (secondary) 


13 


69.2 


Science 


6 


50.0 


Resource 


4 


100.0 


Other 


9 


90.0 


Eocation 


Idaho 


33 


93.4 


Pennsylvania 


9 


44.4 


Elorida 


5 


20.0 


Other 


5 


60.0 



Source: Mathematica Policy Research, 2006 Survey of Principals of Passport to Teaching 

Teacher. 
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Not surprisingly, the respondents are fairly representative of the full sample (universe) of 
eligible Passport teachers. Table II.2 shows the characteristics of respondents, nonrespondents, 
and the full sample. The respondents are within 7 percentage points of the full sample on each 
subgroup defined by school type, grade level, and subject area. The largest difference between 
the respondent sample and the universe is the percentage from Idaho, with the state somewhat 
over-represented in our survey sample, and the other states, mainly Florida and Pennsylvania, 
each under-represented. 



TABLE II.2 

CHARACTERISTICS OE RESPONDENTS AND NONRESPONDENTS (PERCENTAGES) 



Subgroup 


Pull Sample 


Respondents 


Nonrespondents 


School Type 


Traditional public 


73.1 


71.8 


76.9 


Charter 


7.7 


10.3 


0.0 


Private 


19.2 


17.9 


23.1 


Grade Eevel 


Elementary 


32.7 


38.5 


15.4 


Middle 


9.6 


10.3 


7.7 


High 


57.7 


51.3 


76.9 


Teaching Assignment 


Elementary (all subjects) 


19.2 


23.1 


7.7 


English/language arts (secondary) 


19.2 


15.4 


30.8 


Mathematics (secondary) 


25.0 


23.1 


30.8 


Science 


11.5 


7.7 


23.1 


Resource 


7.7 


10.3 


0.0 


Other 


17.3 


20.5 


7.7 


Eocation 


Idaho 


63.5 


79.5 


15.4 


Pennsylvania 


17.3 


10.3 


38.5 


Elorida 


9.6 


2.6 


30.8 


Other 


9.6 


7.7 


15.4 



Source: Mathematica Policy Research, 2006 Survey of Principals of Passport to Teaching 

Teacher. 
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The resulting sample included principals with a range of experience and a range of time 
spent observing their teachers. Principals’ average level of experience in the classroom as a 
teacher was 12 years, and their average experience as an administrator was 8 years, with a range 
of 1 to 26 years. The average principal’s duration of a teacher observation was 36 minutes, and 
most principals reported that they observed the teacher at least three times during the year. 

The most surprising result was the experience level of the Passport teachers themselves, 
many more of whom than expected were already teaching before the 2005-2006 school year. 
We asked only whether the teacher was a first-year teacher, but most respondents volunteered 
the teachers’ years of experience in the profession or in the school, allowing us to estimate the 
distribution of experience in the sample. Table II.3 shows the principal-reported experience 
level of Passport holders who were teaching in their schools. Only 31 percent were in their first 
year. Of the remainder, many were in the early part of their careers, but several were veteran 
teachers, with six or more years of experience. Presumably, these teachers sought the American 
Board’s Passport certification as a means to prove subject area mastery and not to meet initial 
licensure requirements to enter the teaching profession. 



TABLE II.3 

EXPERIENCE LEVEL OE PASSPORT TEACHERS 



Years of Teaching Experience Before 2005- 
2006 


Number of Passport Holders 


Percentage 


None 


12 


30.8 


1-5 


11 


28.2 


6-10 


2 


5.1 


More than 10 


5 


12.8 


1 or more, number unknown 


9 


23.1 


Total 


39 


100.0 



Source: Mathematica Policy Research, 2006 Survey of Principals of Passport to Teaching Teacher. 
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III. FINDINGS 



We present the survey findings in two parts corresponding to two areas of inquiry. The first 
area concerns the role of the Passport certificate in the hiring process. The second area concerns 
the effectiveness of Passport teachers once hired. In addition, we discuss some supplemental 
analyses that we conducted to help readers better understand the survey data. 

A. HIRING DECISIONS AND PERCEPTIONS OF ABCTE 

Overall, the respondents responsible for hiring the teachers in our sample had positive 
assessments of those teachers but were more moderate in their opinions of ABCTE certification, 
when they were aware of it at all. 

The Passport holders had generally been hired in a competitive environment. Almost all of 
them competed against other candidates for their positions (see Table III.l). If the position was 
competitive, we asked the respondent to compare the Passport holder who filled the position to 
the other candidates overall. Three-quarters of respondents ranked the Passport holder 

“somewhat” or “much” stronger than the other applicants. Only one teacher was rated 

2 

“somewhat weaker” than other candidates. 

An interesting finding — likely related to the high proportion of our sample with teaching 
experience — is that the majority of Passport holders received their job offer even before they 
obtained a Passport (19 of 30, or 63 percent). Many of these teachers had been hired several 
years earlier, in which case the Passport was often a means to certify their subject matter 
expertise rather than help them gain entry into the profession. A high proportion of respondents, 
almost 20 percent, did not know if the candidate had ABCTE certification at the time of hiring. 

^ We did not interview principals who declined to hire Passport holders, so we cannot assess whether 
certification was a factor in unsuccessful job candidacies. 
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For those teachers who did have that certification, their hirers were fairly evenly split between 
indicating that ABCTE certification had no impact or a positive impact. None indicated that 
ABCTE certification had a negative impact. 



TABEEIII.l 

EACTORS REEATED TO HIRING DECISIONS 



Eactors 


Number 


Percentage 


Hiring decision was competitive 


33 


90.3 


How ABCTE candidate compared to other candidates 


Much weaker 


0 


0.0 


Somewhat weaker 


1 


3.1 


About the same 


7 


21.5 


Somewhat stronger 


8 


24.6 


Much stronger 


17 


50.8 


Teacher obtained ABCTE certification after being hired 


Yes 


19 


51.4 


No 


11 


30.6 


Don’t know 


7 


18.1 


Impact that ABCTE certification had on hiring decision 


Negative impact 


0 


0.0 


No impact 


6 


54.6 


Positive impact 


5 


45.5 


Respondent would hire ABCTE teacher in the future 


Yes 


15 


40.9 


No 


0 


0.0 


Depends 


19 


53.5 


Don’t know 


2 


5.6 



Total 37 



Source: Mathematica Policy Research, 2006 Survey of Principals of Passport to Teaching 

Teacher. 
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Interested in the reasons for their response, we probed those who indicated that the Passport 
had an impact on their hiring decision. One respondent said that he/she “wouldn’t have hired 
[the person] without certification.” Two cited the subject matter certification as the key to hiring 
for the given positions. Another respondent said that obtaining the certification was a signal of 
working harder. “Any time we see teachers go above and beyond, that’s a positive.” 

Given the fairly untested nature of ABCTE certification, it is a good indication of the quality 
of our sample of Passport holders that, when asked if they would hire a teacher certified by 
ABCTE, 40 percent of hirers responded with an unqualified “yes.” This rate was similar for 
respondents who had hired experienced (43 percent) and novice teachers (33 percent). 
Approximately half of all respondents said that their decision to hire another Passport holder 
would depend on other factors, and the remainder said they did not know whether they would 
hire another Passport holder. No respondent indicated that he or she would not hire another 
Passport holder. 

The respondents’ more general comments about ABCTE in response to an open-ended 
question were illustrative and generally positive. Several mentioned that it was their first 
exposure to ABCTE and that the teacher in question had worked out well; one was “skeptical at 
first” but asserted that ABCTE is an “excellent alternative for people who are qualified to be 
good teachers.” Others were advocates of the certification and had encouraged current teachers 
to pursue it, often to meet requirements from the No Child Eeft Behind legislation. Another 
commonly cited benefit of the program was the identification of individuals with expertise in a 
particular subject, often mathematics and science. More than one respondent commented on the 
rigor of the examination. 

Of the less positive responses, the most common criticism was the lack of classroom 
experience. One principal of a Passport holder who received low ratings was concerned that 
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“individuals do not have practical student teaching experience. It is imperative to have 
experience with an accompanying teacher to help with planning, classroom management, etc.” 

B. PERFORMANCE RATINGS 

Respondents gave Passport teachers positive ratings on average for all dimensions of 
performance. As shown in Table III.2, the typical rating was somewhere between “equally as 
effective” as and “somewhat more effective” than other teachers observed by the supervisors in 
their careers on nearly every performance dimension rated. A rating of zero suggests that the 
Passport teacher is “equally effective,” and an average rating of 0.5, for example, means that the 
responses are halfway between “equally” and “somewhat more” effective.^ 

The first column in Table III.2 shows that the Passport teachers overall were rated closer to 
“somewhat more effective” than “equally effective,” with an average rating of 0.7. The highest 
ratings were in “communicating content knowledge” (0.9) and “responding to feedback from 
other teachers and administrators” (1.0) and the lowest in “managing the classroom” (0.3). 

If the Passport holder in question was a first-year teacher, we also asked the supervisor to 
compare that teacher to other first-year teachers they had observed over their career. The second 
column in Table III.2 shows the novice-to-novice comparison. The last two columns in Table 
III.2 are subgroups (novices and experienced teachers) as compared with all other teachers. 

Beginning teachers with Passport certificates fared well when compared with other first- year 
teachers, scoring closer to “somewhat more effective” than “equally effective” (0.6). They rated 
highest in “engaging students in learning” (0.8) and lowest in “adapting instruction” (0.3) and 
classroom management (0.3) as compared with other first-year teachers. Not surprisingly, the 



This method treats the interval between each response option on the five-point scale as the same. Given that 
respondents rarely used the extreme options, the results vary little when we relax this assumption. 
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TABLE III.2 



EFFECTIVENESS OF PASSPORT HOLDERS AS RATED BY THEIR PRINCIPALS 





All Passport 
Teachers 


Passport Holders in First Year 
of Teaching 


Passport Holders with 
Two or More Years of 
Teaching Experience 


Area of Effectiveness 


Compared 
with All Other 
Teachers 


Compared 
with Other 
First-Year 
Teachers 


Compared 
with All 
Other 
Teachers 


Compared with All 
Other Teachers 


Understanding students’ needs 


0.6 


0.5 


0.3 


0.8 


Planning lessons 


0.5 


0.5 


-0.2 


0.8 


Leading instructional activities 


0.8 


0.5 


0.3 


1.0 


Adapting instruction 


0.8 


0.3 


0.3 


1.0 


Managing the classroom 


0.3 


0.3 


-0.3 


0.6 


Encouraging desired student behavior 


0.6 


0.4 


0.1 


0.9 


Engaging students in learning 


0.7 


0.8 


0.3 


0.9 


Communicating content knowledge 


0.9 


0.6 


0.1 


1.3 


Collaborating with other teachers 


0.8 


0.5 


0.3 


1.0 


Responding to feedback from other 
teachers and administrators 


1.0 


0.7 


0.7 


1.2 


Relating to parents 


0.6 


0.5 


-0.2 


1.0 


Overall performance 


0.7 


0.6 


0.3 


0.9 


Sample size (number of teachers) 


39 


12 


12 


27 



Source: Mathematica Policy Research, 2006 Survey of Principals of Passport to Teaching Teachers. 

Note: Relative teacher effectiveness was rated on a five-point scale: 

-2 = Much less effective 
-1 = Somewhat less effective 

0 = Equally effective 

1 = Somewhat more effective 

2 = Much more effective 

Cell entries are average ratings. 

Ratings are weighted to account for multiple respondents for one teacher. 



14 




beginning teachers’ ratings fell in nearly every category when compared with teachers of all 
experience levels, but their overall rating was still positive (0.3). In these comparisons, 
beginning teachers were rated lowest in “planning lessons” (-0.2), classroom management (-0.3), 
and “relating to parents” (-0.2). They were rated highest in responding to feedback (0.7). 

Passport holders with teaching experience fared better than Passport holders new to 
teaching, with an overall performance rating of 0.9. In the individual categories, they rated 
highest in communicating content knowledge (1.3) and responding to feedback (1.2). 

Table III.3 shows the distribution of ratings across the full five -point scale for the overall 
performance measures and for two other questions about American Board teachers and 
certification. According to the overall ratings, the most common responses were that Passport 
holders’ overall performance as teachers was “average” or “above average” (see Figure III.l). 
Similarly, for the individual performance items noted in Table III.2, the most common responses 
fall in the middle three categories. The rare exceptions are two teachers who were rated 
“somewhat less effective” or “much less effective” on nearly every measure, which translates 
into a rate of 5 percent who were underperformers. The other 95 percent were judged as 
effective or better than their peers. 

For first-year teachers, we asked principals to tell us if they are confident in their first- year 
teachers’ ability to develop into an effective teacher. All but one reported being “confident” or 
“very confident” about this outcome. 

We also asked respondents if their experiences gave them confidence in the American Board 
certification. None of the principals reported “a great deal of confidence” in the certification 
based on teacher performance, but nearly half reported “some confidence,” nearly half reported 
“a little confidence,” and the remaining 6 percent (two respondents) said that they had “no 
confidence,” corresponding to the two respondents who rated their teachers as ineffective in 



15 




every area. The moderate nature of the responses to the confidence question suggests that 



principals are probably reluctant to generalize from the experience of a single teacher. 



TABLE III. 3 

ERWCIPAL RATEsfGS OF OVERALL PERFORMANCE 
AND CONFIDENCE IN TEACHERS AND ABCTE 



Rating 


Number 


Percentage 


Overall performance as compared with all teachers 
Substantially below average 


1 


2.6 


Below average 


3 


7.7 


Average 


12 


30.8 


Above average 


13 


32.1 


Substantially above average 


11 


26.9 



Overall performance as compared with first-year teachers 
(first-year Passport teachers only) 



Substantially below average 


1 


8.3 


Below average 


1 


8.3 


Average 


2 


16.7 


Above average 


6 


50.0 


Substantially above average 


2 


16.7 



Confidence that Passport holder will develop into an effective 
teacher, confidence as compared with other first-year teachers 
(first-year Passport teachers only) 



Not at all confident 


1 


8.3 


Somewhat confident 


0 


0.0 


Confident 


4 


33.3 


Very confident 


7 


58.3 



Confidence in ABCTE certification, based on teachers’ 
performance 



No confidence 


2 


5.6 


A little confidence 


15 


41.7 


Some confidence 


16 


43.1 


A great deal of confidence 


0 


0.0 


Don’t know 


4 


9.7 



Source: Mathematica Policy Research, 2006 Survey of Principals of Passport to Teaching Teachers. 
Note: Percentages are weighted to account for multiple respondents for one teacher. 
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FIGURE III.1 



PRINCIPAL RATING OF OVERALL PERFORMANCE OF ABCTE 
TEACHERS AS COMPARED WITH ALL TEACHERS 



Number of 
Teachers 




Substantially Below average Average Above average Substantially 
below average above average 

Principal Rating 



Source: Mathematica Policy Research, 2006 Survey of Principals of Passport to Teaching Teachers. 



C. UNDERSTANDING THE PERFORMANCE RATINGS 

To help interpret the findings in the previous sections, we looked for evidence in our data on 
whether ratings might reflect something other than teacher effectiveness and whether principals 
place implicit weights on some dimensions of performance over others. 

First, ratings vary significantly by the context in which the survey was administered — 
specifically, the characteristics of the respondent and his or her school or the characteristics of 
the Passport holder. We used a regression model to estimate the relationship between the ratings 
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and the following characteristics:"^ principal’s years of experience as an administrator, the 
amount of time the principal reported observing the teacher in the classroom, whether the teacher 
was in his or her first year of teaching, the teaching assignment (subject area), and the type of 
school — public, private, or charter (see Table 111.4).^ 

In general, differences in ratings by teaching assignment were statistically significant, but 
differences in teacher and principal experience were not. Teachers in charter schools received 
higher ratings than those in traditional public schools while those in private schools received 
lower ratings (although not statistically significant). Elementary school teachers received lower 
ratings than teachers in all other subjects, particularly resource teachers and those teaching 
secondary mathematics or language arts. We also estimated models with each component as the 
dependent variable and found similar results. 

Another issue is that, while we report the principal’s overall rating of a Passport holder, it 
may not be obvious that (or how) the overall rating is an aggregate measure reflective of the 
individual components of effectiveness. To try to gauge this, we used another regression model 
to estimate the relationship between the overall rating and the 1 1 individual performance areas 
(see Table III.5). The coefficients were not statistically significant for any of the components, 
which is not surprising given the sample size and collinearity of the components. Nevertheless, 
we can still discern some trends in the relative weight given each measure without generalizing 
beyond this sample. The components with the largest coefficients (and therefore, presumably, 
the largest contribution to the overall rating) were engaging students in learning, understanding 

“ We reported findings from a linear regression model that treats the responses on the five-point scale as 
cardinal with equal intervals. We also estimated an ordered logit model, which relaxes the equal interval 
assumption. We did not report the ordered logit results, but they led to the same conclusion. 

^ Although there were several ways we could include the relevant data for time observed (e.g., as the total 
duration of observations versus separate variables for frequency and average length), the model was not sensitive to 
the way in which we defined that measure. 
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students’ needs, and classroom management. Least important were encouraging desired student 



behavior, responding to feedback, and leading instructional activities. 



TABLE III.4 

THE EEEECT OE PRINCIPAE AND TEACHER CHARACTERISTICS 

ON OVERAEE RATINGS 



Characteristic 


Coefficient 


Standard Error 


P-value 


Teacher is a novice (first year) 


0.057 


0.398 


0.887 


Supervisors’ experience level (years) 


0.034 


0.025 


0.183 


Time supervisor reports observing teacher 


(minutes) 


0.000 


0.001 


0.783 


School Type 


Public, traditional 


0.000 


n.a. 


n.a. 


Public, charter 


1.641 


0.575 


0.008*** 


Private 


-0.369 


0.481 


0.448 


Teachers’ Assignment 


Elementary education 


0.000 


n.a. 


n.a. 


Secondary English/Eanguage Arts 


0.977 


0.574 


0.100 


Secondary Mathematics 


0.961 


0.539 


0.085** 


Resource teacher (any grade) 


1.380 


0.699 


0.058** 


Science 


0.492 


0.640 


0.448 


Other 


0.543 


0.510 


0.295 


Intercept 


2.705 


0.638 


0.000*** 


R- squared 

Sample Size (teachers) 




0.24 

39 





Source: Mathematica Policy Research, 2006 Survey of Principals of Passport to Teaching 
Teachers. 

n.a. = not applicable, coefficient on omitted variable not estimated. 

*Significantly different from zero at the .10 level. 

**Significantly different from zero at the .05 level. 

***Significantly different from zero at the .01 level. 
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TABLE III.5 



THE EEEECT OE COMPONENTS ON OVERAEE EEEECTIVENESS RATING 



Explanatory Variable 


Coefficient 


Standard Error 


P-value 


Understanding students’ needs 


0.273 


0.237 


0.259 


Planning lessons 


0.030 


0.165 


0.857 


Eeading instructional activities 


-0.040 


0.197 


0.839 


Adapting instruction 


0.086 


0.162 


0.600 


Managing the classroom 


0.160 


0.145 


0.281 


Encouraging desired student behavior 


-0.082 


0.172 


0.637 


Engaging students in learning 


0.336 


0.224 


0.147 


Communicating content knowledge 


0.131 


0.166 


0.436 


Collaborating with other teachers 
Responding to feedback from other teachers and 


0.067 


0.175 


0.703 


administrators 


-0.047 


0.172 


0.789 


Relating to parents 


0.010 


0.183 


0.957 


Intercept 


0.383 


0.324 


0.248 


R- squared 




0.82 




Sample Size (teachers) 




36 





Source: Mathematica Policy Research, 2006 Survey of Principals of Passport to Teaching 

Teachers. 



20 




IV. CONCLUSIONS 



A. SUMMARY OF FINDINGS 

The administrators who hire and supervise Passport holders rendered their judgments on the 
first cohort of certificate holders and gave a generally positive assessment of the teachers and a 
cautious assessment of the program that certified them. 

In terms of the teachers themselves, the supervisors’ experiences were generally positive. 
Those respondents who hired the ABCTE teachers said that the teachers were stronger than the 
other job candidates. Those who observed the ABCTE teachers (typically the same respondent) 
said that they were as or more effective than the average teacher they had observed during their 
careers. In terms of specific dimensions of teaching, the principals gave Passport teachers 
slightly stronger ratings on their responsiveness to feedback and their ability to communicate 
content knowledge and weaker ratings on classroom management. 

Respondents also, however, demonstrated some hesitation about attributing the positive 
experiences to the American Board certification. A majority of Passport teachers had already 
been teaching when they received their credential; therefore, the Passport could not have played 
a role in their being hired. Eor those who earned the credential in order to teach, administrators 
did not feel the credential affected the hiring decision, or they could not even recall whether the 
teacher had the credential at the time of hiring. Principals’ confidence in the potential of their 
novice Passport holders to develop into effective teachers did not translate into a high level of 
confidence in the Passport credential. We assume that such a response reflects a reluctance to 
extrapolate from the experience with a single teacher to all other teachers who earned the same 
credential. 
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Our finding that most of the early Passport holders were not new to the teaching profession 
when they got their credential was important for understanding how the program is used. 
Accordingly, we conclude that the early adoption of ABCTE’s Passport to Teaching was a 
means to certify subject matter expertise so that teachers could either move into or keep a 
position in a particular field; early adoption was not usually a means to enter teaching. 

B. LIMITATIONS OF THE STUDY 

This study has some important limitations. It is only the first of many reports on the 
American Board certification programs and includes data on the first cohort of teachers. This 
pioneer group is likely to be idiosyncratic and smaller than future cohorts. Yet, even for the 
current cohort, we are continuing to collect data on student achievement gains so that, in the 
future, we will be able to form more objective measures of performance for consideration along 
with the current findings. 

The performance ratings are subjective. They vary with school characteristics, which could 
mean that they vary for reasons other than teacher quality, although at this point we have no way 
to estimate the relationship independently. With test scores data in hand, we will be able in the 
near future to measure the correlation between principal ratings and achievement growth. Until 
then, we must interpret the findings in light of how we think principals would have rated a 
comparable group of teachers who received traditional certification. For example, principals 
may have a tendency to rate most teachers above average. 

Finally, by focusing only on Passport holders who were teaching, we did not address the 
teaching quality or potential teaching quality of those who delayed their entry into teaching or 
were unable to find teaching positions. We will address this and many other limitations in future 
components of the longitudinal study as outlined in our design report (Glazerman and Tuttle 
2006). For example, principals may have a tendency to rate most teachers above average. 



22 




REFERENCES 



Armor, David, Patricia Conry-Oseguera, Millicent Cox, Nicelma King, Lorraine McDonnell, 
Anthony Pascal, Edward Pauly, and Gail Zellman. “Analysis of the School Preferred 
Reading Program in Selected Los Angeles Minority Schools.” Santa Monica, CA: RAND 
Corporation, 1976. 

Glazerman, Steven, and Christina Tuttle. “American Board Certification: Progress and Plans.” 
Washington, DC: Mathematica Policy Research, Inc., March 2006. 

Jacob, Brian, and Lars Lefgren. “Principals as Agents: Subjective Performance Measurement in 
Education.” NBER Working Paper Number 11463. Cambridge, MA: National Bureau of 
Economic Research, July 2005. 

Medley, Donald, and Homer Coker. “The Accuracy of Principals’ Judgments of Teacher 
Performance.” Journal of Education Research, vol. 80, no. 4, 1987, pp. 242-247. 

Murnane, Richard. The Impact of School Resources on the Learning of Inner-City Children. 
Cambridge, MA: Ballinger Publishing, 1975. 

Peterson, Kenneth. Teacher Evaluation: A Comprehensive Guide to New Directions and 
Practices (Second Edition). Thousand Oaks, CA: Corwin Press, 2000. 

Peterson, Kenneth. “Teacher Evaluation with Multiple and Variable Lines of Evidence.” 
American Education Research Journal, vol. 24, no. 2, 1987, pp. 311-317. 



23 




